Searching for user interface objects

ABSTRACT

A computer-implemented method includes searching resources related to an application in response to receiving a query, identifying an operation in the application that satisfies the query, and exposing an object that provides access to the identified operation, within the application.

CROSS REFERENCE TO RELATED APPLICATION

This application is a continuation of, and claims the benefit ofpriority under 35 USC 120 to, U.S. patent application Ser. No.11/857,347, filed Sep. 18, 2007, titled “SEARCHING FOR USER INTERFACEOBJECTS,” the disclosure of which is considered part of the disclosureof this application, and is incorporated herein by reference.

TECHNICAL FIELD

This disclosure relates to information retrieval.

BACKGROUND

Software applications installed on systems, e.g., computer systems,enable users to perform a variety of operations ranging from simple textentry to complicated computations. Some applications, e.g., applicationsin the Microsoft® Office suite offered by Microsoft Corporation(Redmond, Wash.), are visible to the user and enable user interactionthrough a user interface that is provided to the user when anapplication is launched. Other applications, e.g., virus scanningapplications, operate in the background and do not always require userinteraction. Applications that offer a user interface can include one ormore menus including selectable objects that represent operations that asoftware application can perform upon being selected by a user. Anoperation can be represented by a corresponding title containing text oricon containing an image, or both, and displayed in the user interface.

SUMMARY

In one example, a user can enter a query in a text box displayed in auser interface of an application. e.g. Microsoft® Office PowerPoint®,offered by Microsoft Corporation (Redmond, Wash.). The query can berelated to an operation that the application can perform, insertingpictures from a scanner or camera into a slide. An object correspondingto the operation made available through the user interface, e.g., as achoice in a menu of operations, can be identified to the user inresponse to receiving the query. The application's resources, e.g.,program files related to the application, can be searched and thelocation of the object corresponding to the operation can be identified.Subsequently, the location of the object, e.g., the menu where theobject resides, can be exposed, e.g., displayed in the user interface,such that the object becomes apparent to the user. In addition, thesearched operation can be performed automatically—that is, withoutreceiving further input from the user.

In one aspect, a computer-implemented method includes searchingresources related to an application in response to receiving a queryfrom a user, identifying an operation in the application that satisfiesthe query, and exposing an object, within the application, that providesaccess to the identified operation.

This, and other aspects, can include one or more of the followingfeatures. Exposing the object can include causing a location, where theobject resides, to be displayed. The method can further includehighlighting the object. The location can include one or more of a menu,a tab, a dialog box, a sub-menu, a sub-tab, or a sub-dialog box. Themethod can further include identifying content that satisfies the query,wherein the content is unrelated to the operation, and presenting acontent location of the identified content. The method can furtherinclude receiving the query from the application, wherein the query ispresented in a user interface of the application.

In another aspect, a medium bearing instructions to enable one or moremachines to perform operations is described. The operations includesearching resources related to an application in response to receiving aquery from a user, identifying an operation in the application thatsatisfies the query, and exposing an object, within the application,that provides access to the identified operation.

This, and other aspects, can include one or more of the followingfeatures. Exposing the object can include causing a location, where theobject resides, to be displayed. The operations can further includehighlighting the object. The location can include one or more of a menu,a tab, a dialog box, a sub-menu, a sub-tab or a sub-dialog box. Theoperations can further include identifying content that satisfies thequery, wherein the content is unrelated to the operation, and presentinga content location of the identified content. The operations can furtherinclude receiving the query from the application, wherein the query ispresented in a user interface of the application.

In another aspect, a system comprising an engine is described. Theengine is configured to search resources related to an application inresponse to receiving a query from a user, identify an operation in theapplication that satisfies the query, and expose an object within theapplication, that provides access to the identified operation.

This, and other aspects, can include one or more of the followingfeatures. The engine can further be configured to cause a location,where the object resides, to be displayed. The engine can further beconfigured to highlight the object. The location can include one or moreof a menu, a tab, a dialog box, a sub-menu, a sub-tab, or a sub-dialogbox. The engine can further be configured to identify content thatsatisfies the query, wherein the content is unrelated to the operation,and present a content location of the identified content. The engine canfurther be configured to receive the query from the application, whereinthe query is presented in the user interface of the application.

In another aspect, a computer-implemented method is described. Themethod includes searching resources related to an application inresponse to receiving a query, identifying an operation in theapplication that satisfies the query; and automatically performing theoperation.

This, and other aspects, can include one or more of the followingfeatures. The method can further include automatically performing theidentified operation without further input from the user. The method canfurther include identifying content that satisfies the query, whereinthe content is unrelated to the operation, and presenting a location ofthe identified content. The method can further include receiving thequery from the application, wherein the query is presented in a userinterface of the application.

The systems and techniques described here may provide one or more of thefollowing advantages. Enabling a user to search for locations of objectsrepresenting operations that can be performed by an application canallow the user to exploit the features of an application by identifyingoperations that may not be apparent to the user. Automaticallyperforming an operation upon identifying an operation for which the useris searching can enhance user experience. Identifying operations that anapplication enables a user to perform in response to receiving a querycan help educate a user about the features of the application. Enablinga user to search for operations can allow the user to quickly find andexecute operations which may be difficult or non-obvious to otherwisefind. Further, users can find operations in the application withoutknowing or caring about the navigational structure of the application.

The details of one or more implementations are set forth in theaccompanying drawings and the description below. Other features andadvantages will be apparent from the description and drawings, and fromthe claims.

DESCRIPTION OF DRAWINGS

FIG. 1 is a schematic of an example of a system for searching forresources related to an application.

FIG. 2 is a schematic of an example of a user interface of anapplication.

FIG. 3 is a schematic of an example of a user interface of anapplication.

FIG. 4 is a schematic of an example of a system for searching foroperations related to an application.

FIG. 5 is a flow chart of an example of a process for searching for anoperation related to an application.

FIG. 6 is a flow chart of an example of a process for searching for anoperation related to an application.

DETAILED DESCRIPTION

FIG. 1 depicts an example of a schematic of a system 100 for searchingfor resources related to an application. The system 100 can include anengine 105 configured to enable a user to interact with an applicationinstalled on the engine 105. The system 100 can include a display device110 operatively coupled to the engine 105, which can display a userinterface related to the installed application. The system 100 caninclude input devices, e.g., a keyboard 115, and a pointing device 120,e.g., a mouse, operatively coupled to the engine 105 which the user canuse to interact with the application, where the results of theinteraction can be displayed, e.g., in the user interface related to theapplication. In some implementations, the engine 105 can be operativelycoupled to a remote engine 125 over a network 130, where the remoteengine 125 can be, e.g., a host of a web site, e.g., www.google.com, andthe network 130 can be the Internet. In some implementations, the engine105, the display device 110, and the input devices 115 and 120, can bepart of a computer system, e.g., a desktop computer, a laptop computer,a personal digital assistant (PDA), and the like. The application can beany software application, e.g., Microsoft® Office PowerPoint®, that canbe installed on the engine 105. The application can include applicationresources, e.g., program files related to the application, which, wheninstalled on the engine 105, can be executed by the engine 105 toperform operations such as displaying a user interface on the displaydevice 110, receiving input from a user via the input devices, producingoutput that can be displayed in the user interface on the display device110, performing operations in the background that are not visible on thedisplay device 110, and the like. The engine 105 can be operativelycoupled to the remote engine 125 through the network 130 via wired orwireless interfaces.

An application can be installed on the engine 105, e.g., using a CD-ROMincluding the application or by downloading the application from theremote engine 125 over the network 130. Resources, e.g., program files,related to the application can be made available to the engine, e.g., bestored on the engine 105, such that the engine 105 can execute, e.g.,instructions included in the resources, to cause a user interface to bedisplayed on the display device 110 operatively coupled to the engine105, and an object, e.g., an icon, to be displayed on the display device110. In addition, the resources can include instructions to enable auser to perform operations using the application. A user can use aninput device, e.g., the keyboard 115 or the mouse 120 or both, to selectthe object representing the application. The engine 105 can detect thatthe user has selected the object and, in response, can launch theapplication, where, in some implementations, launching the applicationcan include the engine 105 displaying a user interface related to theapplication on the display device 110 and enabling the user to interactwith the user interface using the input devices, e.g., the keyboard 115,the mouse 120, and the like. In some implementations, the applicationcan allow the user to perform several operations, e.g., add content,modify content, delete content, and the like, the results of whichoperations can be viewed in the user interface.

The application user interface can include several objects where eachobject can correspond to one or more operations that the applicationenables the user to perform. For example, opening a file is an operationthat an application enables a user to perform, for which the applicationuser interface can include a corresponding object, e.g., a clickablebutton in a menu, which a user can select. When the user selects theobject, e.g., using the input devices, a default folder is displayed inthe user interface, which the user can use as a starting location toidentify the file that the user wishes to open. Similarly, the engine105 can receive an input from a user through the input devices, wherethe input includes selecting an object corresponding to an operation,displayed in a menu of operations in the user interface of theapplication. In response to the selection, the engine 105 can performthe operation corresponding to the selected object, based on theapplication resources, stored on the engine 105, that define theoperation to be performed.

In some implementations, the engine 105 can display a user interface,related to an application, on the display device 110, in which the usercan enter a query, where the query relates to an operation of theapplication. For example, the user interface can include a text box intowhich a user can enter a query, e.g., a string of text, where the querycan be related to an operation that can be performed by the application.The engine 105 can search the application resources, e.g., program filesrelated to the application, to identify the one or more operations thatsatisfy the query. Subsequently, the engine 105 can identify the objectcorresponding to the searched operation, e.g., the menu where the objectcorresponding to the searched operation resides, and can present theidentified object as a search result. For example, the engine 105 candisplay the menu containing the object corresponding to the searchedoperation and highlight the object. Alternatively, or in addition, theengine 105 can cause the searched operation to be performed.

FIG. 2 depicts a schematic of a user interface 205 of an applicationdisplayed in the display device 110 operatively coupled to the engine105. The engine 105 can display the user interface 205 in response toreceiving a selection from the user to launch the application to whichthe user interface 205 corresponds. The user interface 205 can includeseveral objects 210, 212, 214, 216, 218, 220, 222, and 224 which cancorrespond to operations that the application can enable a user toperform or an identifier for a collection of operations or both. Forexample, object 210 can be titled “File”, wherein object 210 representsa collection of operations, e.g., “Open”, “Close”, “Save”, “Import”, andthe like, where each operation under object 210 can be accessed byselecting object 210. In another example, object 224 can be “Convert toPDF” where selecting the object 224 can cause an operation to beperformed, wherein the operation, in this example, is conversion to PDF.

The user interface 205 can include a text box 230 into which a user canenter a query, e.g., a string of text by positioning the cursor 235,operated by the pointing device 120, in the text box 230, and enteringtext using the keyboard 115. The text can be a query related to anoperation that can be performed by the application. For example, one ofthe operations that can be performed by the application can be titled“3-1”. The object 3-1, corresponding to the operation 3-1, which theuser can select in order to perform the operation 3-1, may not beapparent to the user because the object 3-1 may reside in a menu underobject 214. The user can enter the title “3-1” or text that is relatedto the operation 3-1 in the text box 230 to search for the operation3-1. In response, the engine 105 can identify that the operation 3-1satisfies the query in the text box 230, identify that the operation 3-1is represented by the object 3-1 residing in the menu under object 214,and expose the object 3-1 to the user, e.g., display the menu underobject 214 in the user interface 205 as a search result 240. Inaddition, the engine 105 can highlight the object 3-1, e.g., by fillingthe object 3-1 with a color that can be readily discerned by the user.In this manner, the engine 105 can present the user with the location ofan operation in response to the user's query, and enable the user toselect the operation, e.g., by positioning the cursor 235 on the object3-1 and selecting the object.

In some implementations, the engine 105 can perform the operation 3-1upon identifying that the user was searching for the operation 3-1. Forexample, the application can be Microsoft® Office PowerPoint®, and theoperation can be inserting a picture from a camera or scanner. Theobject that the user needs to select to insert a picture from a cameraor scanner may reside in a menu on the user interface 205, and may,thus, not be apparent to the user. The user can enter “inserting apicture from a camera or scanner” as a query in the text box 230. Theengine 105 can parse the query and identify that the operation that theuser is looking for is inserting a picture from a camera or scanner.Further, by searching the application resources, the engine 105 canidentify that the menu option to enable a user to perform the requiredoperation resides under “Insert→Picture→From Scanner or Camera . . . ”.The engine 105 can cause the hierarchical menu, including the objecttitled “From Scanner or Camera . . . ” to be displayed on the userinterface 205, and, in addition, perform the operation. Thus, a responseto a search for “inserting a picture from a camera or scanner” can bethe engine 105 accessing a scanner or camera operatively coupled to theengine 105 and enabling the user to select a picture located on thescanner or camera by displaying a selection window in the user interface205. Although Microsoft® Office PowerPoint® and inserting a picture fromcamera or scanner as presented as representative examples, the aboveoperation can be performed for any application and for any feature in anapplication. In some implementations, the user can perform the operationwithout any further input from the user. In other implementations, theuser can identify an operation, present the identified operation to theuser, and confirm that the user intended to select the identifiedoperation. If the user confirms that the identified operation is theoperation that the user was searching, then the engine 105 can performthe identified operation. For example, upon identifying an operation,the engine 105 can display a window including the name of the operation,a question asking if the identified operation is the one that the userwas searching, and selectable buttons, such as “Yes” and “No” buttons.The engine 105 can perform the identified operation if the user selects“Yes.”

FIG. 3 depicts a schematic of a user interface 305 of an applicationdisplayed in the display device 110 operatively coupled to the engine105. The engine 105 can display the user interface 305 in response toreceiving a selection from the user to launch the application to whichthe user interface 305 corresponds. The user interface 305 can includeseveral objects 310, 312, 314, 316, 318, 320; 322, and 324 which cancorrespond to operations that the application can enable a user toperform or an identifier for a collection of operations or both. Theuser interface 305 can include a text box 330 in which a user can entera query related to an operation that the application enables a user toperform, content that can be found in, e.g., a data file which the usercreated with the application, a topic related to applicationcapabilities, and the like. When the user enters the query in the textbox 330, the engine 105 can search the application resources to identifyan operation that satisfies the query. In addition, the engine 105 cansearch data files, created using the application, for content thatsatisfies the query and, further, can retrieve content, related to theapplication that satisfies the query, residing on one or more remoteengines 125, connected to the engine 105 through the network 130. Thedata files can reside on one or more storage devices operatively coupledto the engine 105.

For example, the user can enter “3-1” as the query in the text box 330in the user interface 305, where the query can be a string of text. Thestring of text (e.g., “Insert”) can correspond to an operation that canbe performed by the application, e.g., inserting an object. The engine105 can search the application resources to identify the location of theobject containing the text “3-1” that corresponds to the operation thatsatisfies the query “3-1” entered in the text box. In addition, theengine 105 can traverse the contents stored in one or more storagedevices operatively coupled to the engine 105 to identify content, e.g.,data files created using the application, that satisfy the query, e.g.,contain the text “3-1.” Further, the engine 105 can interact with one ormore search engines, e.g., Google search engine, to search one or moreremote engines 125 connected to the engine 105 over the network 130 toidentify content that satisfies the query, e.g., web pages on web sitesthat describe the operation that satisfies the query “3-1.” The one ormore search engines can receive the query from the engine 105 over thenetwork, search network locations, e.g., web pages of web sites, forcontent relevant to the query, and can transfer the results of thesearch to the engine 105. The engine 105 can display the search results340 in the user interface 305 where the results can include the locationof the object “3-1” in the application, the names of one or more datafiles including content that satisfies the query, and identifiers, e.g.,uniform resource locators (URLs), pointing to the location of one ormore network locations that include content that satisfies the query. Insome implementations, the engine 105 can rank the results, where theranking can be based, e.g., on the location that was searched. Forexample, the engine 105 can present application operations that satisfythe query first, followed by content stored on one or more storagedevices operatively coupled to the engine 105, and by content obtainedfrom network locations. The results can further be ranked, e.g., basedon the relevance of each result with respect to the search query. Insome implementations, the universe of content that can be searched,including the application resources, content on storage devices, contentat remote locations, and the like, can be chosen by the user. Forexample, the user can instruct the engine 105 to limit the search toonly application resources. Alternatively, the user can instruct theengine to include content on storage devices and application resourceswhile searching.

FIG. 4 depicts a schematic of an example of a system configured toidentify locations of objects representing operations that anapplication can perform in response to receiving a query related to theoperation. In some implementations, the system can include a displaydevice 110 operatively coupled to the engine 105. The application can beinstalled on the engine 105, where installing the application caninclude making available to the engine 105 application resources 405,e.g., instructions to perform operations in the application, needed tooperate the application, where the application resources 405 can includetextual resources 410 and non-textual resources 415. For example, theapplication resources 405 can be made available to the engine 105 bystoring the textual resources 410 and non-textual resources 425 on theengine 105 when the application is installed and enabling the engine 105to access the application resources 420. The application resources 405can cause the display of the user interface 405 on the display device110 when the user launches the application. Further, when the userselects content displayed in the user interface 420 and an objectrepresenting an operation, e.g., an item titled “Cut” on a menu titled“Edit”, the application resources 425 can cause the engine 105 toperform the “Cut” operation on the content selected by the user. Thetextual resources 410 can include resources relating to objectsdisplaying strings of text in the user interface 420 to operations thatcan be performed by selecting such objects. The non-textual resources415 can include resources relating to images displayed in the userinterface 420 to operations that can be performed by selecting suchimages. Further, in some applications, an object displaying a string oftext and an image can correspond to the same operation. Suchrelationships between objects displaying strings of text, correspondingimages, and corresponding operations can also be stored in applicationresources 405. For example, the application resources 405 can be severaldata files hierarchically stored on a desktop file system, where suchdata files can include the user interface 420, provided by theapplication, that can be displayed on the display device 110 when theapplication is launched.

The engine 105 can launch the application in response to user input,e.g., a selection of an object, such as an icon, displayed on thedisplay device 110 using the keyboard 115 or the mouse 120. The engine105 can retrieve from application resources 405, the user interface 420,and expose the user interface 420, e.g., display the user interface 420on the display device 110. The engine 105 can enable a user to interactwith the user interface 420 using the input devices, and, in response touser inputs, perform operations where the operations performed aredetermined by the application resources 405. In some implementations,the user interface 420 can include a search facility 425 into which theuser can enter a query, e.g., a string of text. The engine 105 caninclude a query parsing module 430 that can receive the query entered inthe search facility 425 and can parse the query, e.g., separate thequery into one or more word phrases based on the query content.Subsequently, the engine 105 can search application resources 405 for anoperation that satisfies the query. For example, the query can be theword “Cut”, which the query parsing module 430 can identify and comparewith textual resources 410 in application resources 405 to identifyinstructions that need to be executed to perform the “Cut” operation.Further, the engine 105 can search application resources 405 to identifythe location, in the user interface, of an object representing the “Cut”operation.

in some implementations, the engine 105 can include an operationexecution module 435 configured to present the identified operation onthe user interface 420. For example, the engine 105 can identify thelocation of the text “Cut” in the application resources 405 and theinstructions to perform the “Cut” operation, and present the locationand the instructions to the operation execution module 435. Theoperation execution module 435 can display, in the user interface 420,the location of the object including the text “Cut”, which the user canselect to perform the cutting operation. In other implementations, inaddition to displaying the location of the object cut in the userinterface 420, the operation execution module 435 can perform theoperation that the user searched for.

FIG. 5 depicts a flow chart of an example of a process 500 foridentifying an operation related to an application in response toreceiving a query. The process 500 can receive a query in a userinterface at 505. For example, the engine 105 can display a userinterface in a display device 110, where the user interface can berelated to an application e.g., a software application such asMicrosoft® Office PowerPoint®, and can be made available to the engine105 when the application is installed in the engine 105. The userinterface can include a text box into which a user can enter a query,e.g., as a string of text.

The process 500 can search application resources to identify anoperation that satisfies the query at 510. For example, when anapplication is installed on the engine 105, application resources,needed to perform the operations of the application, can be stored on astorage device operatively coupled to the engine 105. The applicationresources can include the user interface and instructions to enable theengine 105 to display the user interface in a display device 110 inresponse to receiving instructions from the user. A user interface canbe displayed on the display device 110 into which a user can enter aquery, e.g., a string of text. The engine 105 can search the applicationresources to identify resources that satisfy the received query, e.g.,compare the text in the query with the application resources to identifytext in the application resources that match the text. In this manner,the engine 105 can identify operations that the application enables auser to perform, where the title of the operation satisfies the receivedquery. For example, in response to a query “remove content”, the process500 can search the application resources and identify the operation“Delete”.

The process 500 can expose an object that provides access to theidentified operation 515, e.g., on the user interface on the displaydevice 110. For example, the engine 105 can display the identifiedoperation that satisfied the query on the user interface. The identifiedoperation can be the title of an object that can reside in the userinterface, e.g.; in a menu. In response to receiving the query, theengine 105 can display the menu, in which the object corresponding tothe identified operation resides, in the user interface.

FIG. 6 depicts a flow chart of an example of a process 600 forautomatically performing the operation that is identified upon searchingthe application resources. The process 600 can search applicationresources to identify an operation that satisfies the query at 610. Forexample, when an application is installed on the engine 105, applicationresources, needed to perform the operations of the application, can bestored on a storage device operatively coupled to the engine 105. Theapplication resources can include the user interface and instructions toenable the engine 105 to display the user interface in a display device110 in response to receiving instructions from the user. A userinterface can be displayed on the display device 110 into which a usercan enter a query, e.g., a string of text. The engine 105 can search theapplication resources to identify resources that satisfy the receivedquery, e.g., compare the text in the query with the applicationresources to identify text in the application resources that match thetext. In this manner, the engine 105 can identify operations that theapplication enables a user to perform, where the title of the operationsatisfies the received query. For example, in response to a query“remove content”, the process 600 can search the application resourcesand identify the operation “Delete”.

The process 600 can automatically perform the identified operation at615. For example, if the search query is “make letters bold”, and theengine 105 identifies “Bold” as the operation that satisfies the query,the engine 105 can replace selected text in the user interface with boldtext. In some implementations, the process 600 can automatically performthe identified operation without further input, e.g., from a user. Inother implementations, the process 600 can confirm that the identifiedoperations are the operations that the user was searching for, prior toautomatically performing the identified operation.

Implementations of the subject matter and the functional operationsdescribed in this specification can be implemented in digital electroniccircuitry, or in computer software, firmware, or hardware, including thestructures disclosed in this specification and their structuralequivalents, or in combinations of one or more of them. Implementationsof the subject matter described in this specification can be implementedas one or more computer program products, i.e., one or more modules ofcomputer program instructions encoded on a computer-readable medium forexecution by, or to control the operation of, data processing apparatus.The computer-readable medium can be a machine-readable storage device, amachine-readable storage substrate, a memory device, a composition ofmatter effecting a machine-readable propagated signal, or a combinationof one or more of them. The term “data processing apparatus” encompassesall apparatus, devices, and machines for processing data, including byway of example a programmable processor, a computer, or multipleprocessors or computers. The apparatus can include, in addition tohardware, code that creates an execution environment for the computerprogram in question, e.g., code that constitutes processor firmware, aprotocol stack, a database management system, an operating system, or acombination of one or more of them. A propagated signal is anartificially generated signal, e.g., a machine-generated electrical,optical, or electromagnetic signal, that is generated to encodeinformation for transmission to suitable receiver apparatus.

A computer program (also known as a program, software, softwareapplication, script, or code) can be written in any form of programminglanguage, including compiled or interpreted languages, and it can bedeployed in any form, including as a stand-alone program or as a module,component, subroutine, or other unit suitable for use in a computingenvironment. A computer program does not necessarily correspond to afile in a file system. A program can be stored in a portion of a filethat holds other programs or data (e.g., one or more scripts stored in amarkup language document), in a single file dedicated to the program inquestion, or in multiple coordinated files (e.g., files that store oneor more modules, sub-programs, or portions of code). A computer programcan be deployed to be executed on one computer or on multiple computersthat are located at one site or distributed across multiple sites andinterconnected by a communication network.

The processes and logic flows described in this specification can beperformed by one or more programmable processors executing one or morecomputer programs to perform functions by operating on input data andgenerating output. The processes and logic flows can also be performedby, and apparatus can also be implemented as, special purpose logiccircuitry, e.g., an FPGA (field programmable gate array) or an ASIC(application-specific integrated circuit).

Processors suitable for the execution of a computer program include, byway of example, both general and special purpose microprocessors, andany one or more processors of any kind of digital computer. Generally, aprocessor will receive instructions and data from a read-only memory ora random access memory or both. The essential elements of a computer area processor for performing instructions and one or more memory devicesfor storing instructions and data. Generally, a computer will alsoinclude, or be operatively coupled to receive data from or transfer datato, or both, one or more mass storage devices for storing data, e.g.,magnetic, magneto-optical disks, or optical disks. However, a computerneed not have such devices. Moreover, a computer can be embedded inanother device, e.g., a mobile telephone, a personal digital assistant(PDA), a mobile audio player, a Global Positioning System (GPS)receiver, to name just a few. Computer-readable media suitable forstoring computer program instructions and data include all forms ofnon-volatile memory, media and memory devices, including by way ofexample semiconductor memory devices, e.g., EPROM, EEPROM, and, flashmemory devices; magnetic disks, e.g., internal hard disks or removabledisks; magneto-optical disks; and CD-ROM and DVD-ROM disks. Theprocessor and the memory can be supplemented by, or incorporated in,special purpose logic circuitry.

To provide for interaction with a user, implementations of the subjectmatter described in this specification can be implemented on a computerhaving a display device, e.g., a CRT (cathode ray tube) or LCD (liquidcrystal display) monitor, for displaying information to the user and akeyboard and a pointing device, e.g., a mouse or a trackball, by whichthe user can provide input to the computer. Other kinds of devices canbe used to provide for interaction with a user as well; for example,feedback provided to the user can be any form of sensory feedback, e.g.,visual feedback, auditory feedback, or tactile feedback; and input fromthe user can be received in any form, including acoustic, speech,near-touch input, or tactile input.

Implementations of the subject matter described in this specificationcan be implemented in a computing system that includes a back-endcomponent, e.g., as a data server, or that includes a middlewarecomponent, e.g., an application server, or that includes a front-endcomponent, e.g., a client computer having a graphical user interface ora Web browser through which a user can interact with an implementationof the subject matter described is this specification, or anycombination of one or more such back-end, middleware, or front-endcomponents. The components of the system can be interconnected by anyform or medium of digital data communication, e.g., a communicationnetwork. Examples of communication networks include a local area network(“LAN”) and a wide area network (“WAN”), e.g., the Internet.

The computing system can include clients and servers. A client andserver are generally remote from each other and typically interactthrough a communication network. The relationship of client and serverarises by virtue of computer programs running on the respectivecomputers and having a client-server relationship to each other.

While this specification contains many specifics, these should not beconstrued as limitations on the scope of the disclosure or of what maybe claimed, but rather as descriptions of features specific toparticular implementations of the disclosure. Certain features that aredescribed in this specification in the context of separateimplementations can also be implemented in combination in a singleimplementation. Conversely, various features that are described in thecontext of a single implementation can also be implemented in multipleimplementation's separately or in any suitable subcombination. Moreover,although features may be described above as acting in certaincombinations and even initially claimed as such, one or more featuresfrom a claimed combination can in some cases be excised from thecombination; and the claimed combination may be directed to asubcombination or variation of a subcombination.

Similarly, while operations are depicted in the drawings in a particularorder, this should not be understood as requiring that such operationsbe performed in the particular order shown or in sequential order, orthat all illustrated operations be performed, to achieve desirableresults. In certain circumstances, multitasking and parallel processingmay be advantageous. Moreover, the separation of various systemcomponents in the implementations described above should not beunderstood as requiring such separation in all implementations, and itshould be understood that the described program components and systemscan generally be integrated together in a single software product orpackaged into multiple software products.

A number of implementations have been described. Nevertheless, it willbe understood that various modifications may be made without departingfrom the spirit and scope of the disclosure. For example, in someimplementations, the user interface into which a user can enter thequery can be unrelated to any application and can be designed to enablea user to search for operations in several applications. When the userinterface receives a query, the engine 105 can search applicationresources related to one or more applications installed on the engine105 and identify operations in one or more of the several applicationsthat satisfy the search query. For example, if the user enters “Crop” asa search query in the user interface, the engine 105 can searchapplication resources related to several applications and identify allapplications that enable a user to perform the “Crop” operation.Subsequently, the engine 105 can generate identifiers to each operationcorresponding to each application and provide the generated identifierto the user, wherein, when the user selects an identifier, the userinterface of the corresponding application and the location of theobject representing the operation that matched the user's query can bedisplayed in the display device 110.

In some implementations, in addition to displaying the location ofobjects representing the operations that satisfy the query, the engine105 can display icons representing the operations. For example, the“Open” operation, that enables a user to open a file in an applicationcan also be represented, by an icon, e.g., a half-open manila folder. Ifthe operation that satisfies the query presented in the user interfaceis the “Open” operation, in addition to displaying the location of theobject corresponding to the “Open” operation in the user interface, theengine 105 can also display the icon representing the “Open” operationon the user interface. If the icon is already displayed in the userinterface, the engine 105 can highlight the icon so that it is readilydiscerned by the user. The engine 105 can identify the iconcorresponding to an operation by scanning the application resourcesstored in the engine 105 when the application was installed.

In some implementations, the engine 105 can index the applicationresources to search for operations that satisfy queries quicker and in amore efficient manner. The engine 105 can include a crawler configuredto traverse the application resources, e.g., the data files that aremade available to the engine 105 when the application is installed, tocreate the indexed application resources.

In some implementations, the resources of the engine 105 can be includedwith the application such that when the application is installed, theresources of the engine 105 are automatically installed or are providedto a user as an option to install. In other implementations, the engine105″ and the resources of the engine 105 can be made available as aseparate feature which can be added to any application, e.g., in theform of a plug-in.

In some implementations, the application resources can reside at aremote location, e.g., a host of a web site, to which the engine 105 canbe operatively coupled over the network. In such implementations, when aquery searching for the location of objects representing operations isentered in a user interface, the engine 105 can search the applicationresources residing at the remote location to identify locations ofobjects representing the operations that satisfy the query. Theidentified locations can be displayed on a display device 110operatively coupled to the engine 105.

In some implementations, the universe of resources that is searched inresponse to receiving a search query can be controlled by a user. Forexample, the user can choose to search only application resources inresponse to receiving a search query. In other implementations, the usercan choose to include content stored in one or more storage devicesoperatively coupled to the engine 105 and the content available atremote locations operatively coupled to the engine 105 via the network,in addition to the application resources, in the universe of resourcesthat are searched. In some implementations, the objects representing theoperations can reside in one or more of tabbed windows, dialog boxes,sub-menus, sub-tabbed windows, sub-tabbed dialog boxes, and the like.Accordingly, other implementations are within the scope of the followingclaims.

1. A method comprising: executing a software application on a computingdevice, the software application having a user interface that includes amenu that provides access to operations that the software application isconfigured to perform, wherein at least a portion of the menu is hiddenon the user interface when the menu is visually collapsed; receiving aquery that relates to at least one of the operations; identifying anoperation from the at least one of the operations that is responsive tothe query; identifying, using the computing device, a menu item thatcorresponds to the identified operation, wherein activation of the menuitem causes the identified operation to be performed, and wherein themenu item is located within the hidden portion of the menu when the menuis visually collapsed; and exposing a location within the menu of themenu item, wherein exposing the location comprises visually expanding atleast a portion of the menu to display the menu item.
 2. The method ofclaim 1, wherein exposing the location comprises displaying with theuser interface the location within the menu of the menu item.
 3. Themethod of claim 1, wherein exposing the location comprises highlightingthe menu item.
 4. The method of claim 1, wherein exposing the locationcomprises displaying with the user interface a navigational structure ofthe menu that is used to navigate to the location within the menu of themenu item.
 5. The method of claim 1, wherein the menu is a hierarchicalmenu that includes two or more hierarchical levels of menu items, andwherein exposing the location comprises visually expanding at least aportion of the menu to the hierarchical level at which the menu item islocated.
 6. The method of claim 1, further comprising identifyingelectronic content that is responsive to the query, wherein theelectronic content is related to capabilities of the softwareapplication.
 7. The method of claim 6, further comprising generatingquery results that include the identified menu item and the identifiedelectronic content, and displaying the query results with the userinterface.
 8. The method of claim 7, wherein displaying the queryresults comprises sorting the generated query results based at least inpart on whether the query results correspond to menu items or toelectronic content, and displaying the query results in an order assorted.
 9. The method of claim 7, wherein displaying the query resultscomprises sorting the generated query results based at least in part onrelevance of the query results to the query, and displaying the queryresults in an order as sorted.
 10. The method of claim 1, wherein theuser interface comprises a display device for displaying the menu andone or more input devices for inputting the query.
 11. The method ofclaim 10, wherein the one or more input devices includes a voice inputdevice, and wherein the query is a voice-based query that is receivedvia the voice input device.
 12. A computer-readable storage mediumcomprising instructions that, when executed, cause one or moreprocessors to: execute a software application on a computing device, thesoftware application having a user interface that includes a menu thatprovides access to operations that the software application isconfigured to perform, wherein at least a portion of the menu is hiddenon the user interface when the menu is visually collapsed; receive aquery that relates to at least one of the operations; identify anoperation from the at least one of the operations that is responsive tothe query; identify, using the computing device, a menu item thatcorresponds to the identified operation, wherein activation of the menuitem causes the identified operation to be performed, and wherein themenu item is located within the hidden portion of the menu when the menuis visually collapsed; and expose a location within the menu of the menuitem, wherein exposing the location comprises visually expanding atleast a portion of the menu to display the menu item.
 13. Thecomputer-readable storage medium of claim 12, wherein exposing thelocation comprises displaying with the user interface the locationwithin the menu of the menu item.
 14. The computer-readable storagemedium of claim 12, wherein exposing the location comprises highlightingthe menu item.
 15. The computer-readable storage medium of claim 12,wherein exposing the location comprises displaying with the userinterface a navigational structure of the menu that is used to navigateto the location within the menu of the menu item.
 16. Thecomputer-readable storage medium of claim 12, wherein the menu is ahierarchical menu that includes two or more hierarchical levels of menuitems, and wherein exposing the location comprises visually expanding atleast a portion of the menu to the hierarchical level at which the menuitem is located.
 17. The computer-readable storage medium of claim 12,further comprising instructions that, when executed, cause one or moreprocessors to identify electronic content that is responsive to thequery, wherein the electronic content is related to capabilities of thesoftware application.
 18. The computer-readable storage medium of claim17, further comprising instructions that, when executed, cause one ormore processors to generate query results that include the identifiedmenu item and the identified electronic content, and display the queryresults with the user interface.
 19. The computer-readable storagemedium of claim 18, wherein displaying the query results comprisessorting the generated query results based at least in part on whetherthe query results correspond to menu items or to electronic content, anddisplaying the query results in an order as sorted.
 20. Thecomputer-readable storage medium of claim 18, wherein displaying thequery results comprises sorting the generated query results based atleast in part on relevance of the query results to the query, anddisplaying the query results in an order as sorted.
 21. Thecomputer-readable storage medium of claim 12, wherein the user interfacecomprises a display device for displaying the menu and one or more inputdevices for inputting the query.
 22. The computer-readable storagemedium of claim 21, wherein the one or more input devices includes avoice input device, and wherein the query is a voice-based query that isreceived via the voice input device.
 23. A computing system comprising:a display device; a memory; and a processor that executes a softwareapplication stored in the memory, wherein the software applicationpresents on the display device a menu that provides access to operationsthat the software application is configured to perform, receives a querythat relates to at least one of the operations, identifies an operationfrom the at least one of the operations that is responsive to the query,identifies a menu item that corresponds to the identified operation, andexposes a location within the menu of the menu item, wherein at least aportion of the menu is hidden on the display device when the menu isvisually collapsed, wherein the menu item is located within the hiddenportion of the menu when the menu is visually collapsed, and whereinexposing the location comprises visually expanding at least a portion ofthe menu on the display device to display the menu item.
 24. Thecomputing system of claim 23, wherein exposing the location comprisesdisplaying on the display device the location within the menu of themenu item.
 25. The computing system of claim 23, wherein exposing thelocation comprises highlighting the menu item on the display device. 26.The computing system of claim 23, wherein exposing the locationcomprises displaying on the display device a navigational structure ofthe menu that is used to navigate to the location within the menu of themenu item.
 27. The computing system of claim 23, wherein the menu is ahierarchical menu that includes two or more hierarchical levels of menuitems, and wherein exposing the location comprises visually expanding atleast a portion of the menu on the display device to the hierarchicallevel at which the menu item is located.
 28. The computing system ofclaim 23, wherein the software application further identifies electroniccontent that is responsive to the query, wherein the electronic contentis related to capabilities of the software application.
 29. Thecomputing system of claim 28, wherein the software application furthergenerates query results that include the identified menu item and theidentified electronic content, and displays the query results on thedisplay device.
 30. The computing system of claim 29, wherein displayingthe query results comprises sorting the generated query results based atleast in part on whether the query results correspond to menu items orto electronic content, and displaying the query results in an order assorted.
 31. The computing system of claim 29, wherein displaying thequery results comprises sorting the generated query results based atleast in part on relevance of the query results to the query, anddisplaying the query results in an order as sorted.
 32. The computingsystem of claim 23, further comprising a keyboard input device, whereinthe query is a text-based query that is received via the keyboard inputdevice.
 33. The computing system of claim 23, further comprising a voiceinput device, wherein the query is a voice-based query that is receivedvia the voice input device.
 34. A method comprising: executing asoftware application on a computing device, the software applicationpresenting a menu that provides access to operations that the softwareapplication is configured to perform, wherein at least a portion of themenu is hidden when the menu is visually collapsed; receiving a querythat relates to the software application; identifying an operation fromthe operations that the software application is configured to perform,wherein the identified operation is responsive to the query;identifying, using the computing device, a menu item that corresponds tothe identified operation, wherein activation of the menu item causes theidentified operation to be performed, and wherein the menu item islocated within the hidden portion of the menu when the menu is visuallycollapsed; and exposing a location within the menu of a menu item thatcorresponds to the identified operation, wherein exposing the locationcomprises visually expanding at least a portion of the menu to displaythe menu item.
 35. The method of claim 34, wherein exposing the locationcomprises displaying the location within the menu of the menu item. 36.The method of claim 34, wherein exposing the location compriseshighlighting the menu item.
 37. The method of claim 34, wherein exposingthe location comprises displaying a navigational structure of the menuthat is used to navigate to the location within the menu of the menuitem.
 38. The method of claim 34, wherein the menu is a hierarchicalmenu that includes two or more hierarchical levels of menu items, andwherein exposing the location comprises visually expanding at least aportion of the menu to the hierarchical level at which the menu item islocated.
 39. The method of claim 34, wherein the query is a text-basedquery.
 40. The method of claim 34, wherein the query is a voice-basedquery.